from sklearn_benchmarks.reporting.hpo import HPOReporting
import pandas as pd
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

Gradient boosting: randomized HPO benchmarks

reporting_hpo = HPOReporting(config="config.yml")
reporting_hpo.make_report()

Here, we assume there is no perfect match of hyperparameters between librairies. We perform a randomized hyperparameters search with a fixed time budget for each library.

Benchmark results for fit

Raw fit times vs. validation scores

Smoothed HPO Curves

We boostrap 10 000 times the hyperparameters optimization data points represented on the plot above. Then we compute the average cumulated time to reach a specific validation score by taking the mean across the bootstrapped samples. The shaded areas represent the quartiles of the bootstrap samples. The fastest libraries are therefore the closest to the upper left corner. The specification of the HP grid can be found in the configuration file.

Speedup barplot

The speedup threshold is found by taking 95% of the best score of the worst performing library.

Speedup curves

Benchmark results for predict

Raw predict times vs. validation scores